Skip to main content

Incident Report: Arbitrum Outage

Date: 2023-12-15
Time: 19:06 (GMT+3)
Duration: 1 hour 14 minutes

Description​

INR/USD on Arbitrum reported a heartbeat exceeded issue, indicating that the data feed was not updated for an extended period. Further investigation revealed that several feeds all feeds on Arbitrum were affected, with particular concern for CNY and INR as they are actively monitored.

Root Cause​

Arbitrum chain's downtime was identified as the primary reason for the data feed disruption. The last block update was reported 38 minutes prior to the incident notice.

Impact​

No impact since the whole chain had an outage.

Timeline​

  • 19:06 - Arda reported the INR/USD heartbeat issue.
  • 19:08 - Burak shared that Arbitrum chain is down.
  • 20:20 - Bedirhan switched the Reblok proxy to point to the public RPC, resolving the Arbitrum chain downtime issue.

Lessons Learned​

Quick adaptation to chain downtimes by switching to alternative sources can minimize the impact.

Actions Taken​

  1. Identification of the heartbeat exceedance for INR/USD and related feeds.
  2. Investigation of the Arbitrum chain status and other affected data feeds.
  3. Clarification of monitoring responsibilities and update of the monitoring documentation.
  4. Temporary switch to public RPC for Arbitrum until the centralized chain resumed block production.

Incident Reviewer(s)​

  • Arda (Reported the initial issue and participated in resolution discussions)
  • Burak (Provided insights and guidance on monitoring)
  • Ali (Assisted in clarification and documentation updates)
  • Ashar Shahid (Updated the monitoring spreadsheet)
  • Bedirhan (Implemented the solution by switching to public RPC)
Last updated on by Warren Anderson